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Annual Technical Report Year One 
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1. SUMMARY 

The aims of this research shortly recapitulated are: 

(1) To study the multidimensional spatio-temporo-chromatic signal processing 
capabilities of visual receptive fields 

(2) To identify the key attributes of multidimensional image signals which are 
sensitive to receptive field filters and how these attributes are transformed for 
encoding beyond the receptive fields. 

Spatio-chromatic coding: 

This work shows that units in the visual system are tuned to principal components of 
real world color natural images and provides insight on the strategy the visual system 
may be using to confront natural imagery. It is also useful for identifying image 
components that are significant for the visual system arid those which it either keeps or 
enhances for later stages of signal representation and analysis. The methods by which 
retinal center/surround R/G type cells can be used to realize this coding system are 
also investigated. . 

Spatiotemporal coding: 

In collaboration with NASA, the properties of a number of natural spatio-temporal 
sequences was measured. We found that the sequences showed regularity in both 
temporal and spatial domains.^ We found a spectrum model which represents .98 of 
the signal energy as separable.\ In addition, we developed an efficient coding model for 
natural time varying imagery in the early visual system.HWe show that M&P channels 
conjoin with tracking eye-movement to make visual system retinal architecture matched 
to basic components of time varying imagery. 

Receptive field filtering: 

A new method to analyze parallel and hierarchic processing was developed to account 
for how receptive fields in the visual system combine in terms of their anatomical 
convergence and divergence. 


2. STATUS OF THE RESEARCH 
( i) Spatio Chromatic Coding: 

Neurophysiology provided color research with a wealth of single unit recordings 
indicating how color is coded in the early visual system. These show that when color is 
coded in the visual system it is encrypted together with spatial features. Filters 
(receptive fields) in the visual system have spatial responses with varying spatial 
frequencies which are modulated by different color (wavelength) profiles. Some 
combinations occur frequently in the visual system and some are rare or non-existent. 
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An obvious requirement of the coding system is that it be sufficient to contain all image 
information relevant to the visual system for processing at higher levels of the visual 
system. One of the most commonly applied hypotheses to understand the coding 
strategy is that the visual system is an efficient coder and that it is matched to its signal 
environment which is natural color imagery. By decomposing natural imagery to its 
building blocks we can compare how these are matched by the building blocks of the 
visu&i .mage coding scheme. 

The principal components (eigenfunctions) of a group of signals can serve as a basis 
for that group. The advantages of representing a signal by principal components can 
be briefly summarized as follows, (i) The basis functions of the representation are 
orthogonal and have uncorrelated coefficients, (ii) The expected square coefficients 
(eigenvalues) are minimum compared to other generalized Fourier representations. 
These properties make principal components attractive for efficient signal coding and 
representation, because uncorrelatedness reduce^.' redundancy and smaller 
coefficients require smaller dynamic ranges and hence, less transmission resources. 
Similar considerations have been used before in identifying the strategy of visual 
system operations. The eigenfunctions of black and white images are well-known. 
They include functions with periodic profiles in both spatial dimensions. We extended 
the analysis of the one-plane black and white images to 3-plane color images. 

We computed the principal components of a number of natural color images using 
small sections of the images. We used 3-D blocks of 4 x 4 pixels in each color plane. 
The visual system also applies receptive fields which cover small sections of the image. 
Principal components computed for the natural color images are comprised of basis 
functions with various spatial and color profiles. These principal components are 
similar to certain visual system image operations in space and in color, including the 
color opponent transformation and a variety of spatially organized receptive field filters 
with different orientations. Color opponent profiles are most prominent in low spatial 
frequencies and have very low eigenvalues for high spatial frequencies. For example, 
the fraction of signal energy represented in low spatial frequency color profiles is 
significant, while the expected signal energy In color profiles with high spatial frequency 
is negligible. This suggests that color and space are nearly separable when small 
sections of natural images are considered. Tbis separability diminishes as the size of 
the sections increase. Another advantage of the small sections is their computational 
load. The computational load associated with whole images or large sections thereof, 
in this kind of analysis, is immensely large. We reconstructed the image using the 
eigenfunctions. We find that the image can be reconstructed from a small number of 
eigenfunctions which are similar to visual system receptive fields. 

( ii) Soatio Temporal Coding: 

We investigated the mechanisms and strategy used by the visual system to encode 
time varying imagery. The underlying hypothesis is that the early visual system 
efficiently codes natural time varying imagery. This is accomplished by combining 
tracking, implemented with smooth pursuit eye movements, to prefilter the image and 
shift the image spectrum into a low temporal frequency range and by matching retinal 
pathway characteristics to the properties of imagery after modification by trackino. We 
formulate a representation for time varying imagery which consists of two 
spatiotemporal components. The first component results from 3-D motion in the visual 
scene and is modeled as translational motion (a velocity field). The second component 
results from stationary (nontranslational) temporal changes such as flicker. We show, 
using sequences of natural imagery, that the spatiotemporal spectrum and other 
attributes of natural imagery markedly differ before and after tracking. In the region of 





tracking, the temporal frequency bandwidth and velocity distribution of the translational 
component is diminished. With increasing eccentricity from the region of tracking, the 
temporal frequency bandwidth and velocity distribution broadens. On the other hand, 
the stationary component is unaffected by tracking. Comparison of the proi^erties of 
. the tracked imagery to those of retinal architecture suggests that M and P pathways 
/ logically divide the tracked image spectrum and each transmits different attributes of 
the image. A specialized retinal architecture with a high acuity fovea which fixates on 
1 the tracked region and a decreasing acuity with eccentricity also matches the 
A properties of the tracked signal as they vary with eccentricity from the region of 
tracking. 

(iii) Receptive Field Filterinq: 

The term multirate filter bank (MRB) is used in the filtering literature to describe a group 
of filters with different sampling rates; these can be arranged both in parallel and in a 
hierat chy. The retina is, for all practical purposes, a multirate filter bank. We derived 
general conversions between equivalent parallel and hierarchic muitirate filter banks as 
well as sufficient conditions for existence and uniqueness of the conversions. 

Scaled Gaussian analysis multirate filter banks (MRBs) are analysis MRBs whose filters 
are Gaussians scaled differently in width and height. They are frequently employed in 
vision research as weli as in image processing. We define generalized scaled Gaussian 
analysis MRBs for both parallel and hierarchic architectures and derive conversions 
between them. We calculate the number of multiplications, number of additions, and 
total number of filter coefficients for both architectures implemented in direct form and 
time varying filter implementation. Our results are for both one-dimensional MRBs and 
two-dimensional MRBs with separable filters. In all cases, as has been shown for 
I particular MRBs, the parallel MRB requires considerably more multiplications, additions 
/ and filter coefficients than the equivalent hierarchic MRB. We are now applying this 
new theory to the retinal architecture using anatomical constraints. 
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